Visual annotations on medical images

ABSTRACT

The present disclosure is related to visual annotations on medical images. A system may include a processor configured to process input data and identify a relationship amongst received input data in a data set. The system may also include an aggregator coupled to the processor and configured to receive processed data from the processor and aggregate data within the data set while maintaining one or more data relationships within the data set. Further, the system may include an annotation service module coupled to the aggregator and configured to generate at least one annotation that is maintained across at least a portion of the data within the data set.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/056,817, filed Feb. 29, 2016, which claims the benefit of andpriority to U.S. Provisional App. No. 62/126,206, filed Feb. 27, 2015,each which is incorporated herein by reference in its entirety for allthat it discloses.

TECHNICAL FIELD

This disclosure relates generally to visual annotations on medicalimages. yet more specifically, this disclosure relates to establishingand maintaining data relationships within data sets including data fromvarious data sources. Even more specifically, this disclosure relates toimage annotation and establishing and maintaining data relationshipsacross various imaging modalities and specialties.

BACKGROUND

It is common practice to include annotations (e.g., metadata) with animage (e.g., a digital image) to convey information in a human-readableformat as well as structured information that can be consumed by anexternal system. More specifically, annotations, such as, a region ofinterest, straight line, arrow, measurement results, graphical patterns,characters, and symbols, may be displayed in or near a visible image.

Conventional imaging systems, which may lack context, are complicateddue to data residing in disparate systems from which it is generated,especially when the images are accessed by geographically disconnectedusers. Conventionally, images and related text from a single specialty(e.g., single modality) are maintained in separate systems and are notmaintained in a single collection for use in decision-making. Tools forconveying important information and for linking images from one modalityto images from a different modality do not exist. Moreover, systems andmethods for joining specific areas of interest with annotated regions ofinterest from individual images from the different modalities do notexist.

While there are standards that define how images are generated fordifferent individual specialties and modalities, standards thataggregate and maintain relationships between such images and relatedtextual information in a human and/or machine-readable format do notexist.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one example technology area where some embodiments describedherein may be practiced.

BRIEF SUMMARY

According to one embodiment of the present disclosure, a system mayinclude a processor configured to process input data and identify arelationship amongst input data in a data set. The system may alsoinclude an aggregator coupled to the processor and configured to receiveprocessed data from the processor and aggregate data within the data setwhile maintaining one or more data relationships within the data set.Further, the system may include an annotation service module coupled tothe aggregator and configured to generate at least one annotation thatis maintained across at least a portion of the data within the data set.

According to others embodiments, the present disclosure includes methodsfor processing, aggregating, annotating and/or organizing data. Variousembodiments of such a method may include receiving a plurality of datainputs. The method may also include establishing one or more data setsincluding the plurality of data inputs, wherein one or more data inputsin each data of the one or more data set shares a common identifier.Further, the method may include maintaining relationships amongst datainputs in each data set of the one or more data sets. The method mayalso include generating one or more outputs based on data in a data setof the one or more data sets.

Yet other embodiments of the present disclosure comprisecomputer-readable media storage storing instructions that when executedby a processor cause the processor to perform instructions in accordancewith one or more embodiments described herein.

Other aspects, as well as features and advantages of various aspects, ofthe present disclosure will become apparent to those of skill in the artthrough consideration of the ensuing description, the accompanyingdrawings and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart depicting a method, in accordance with anembodiment of the present disclosure.

FIG. 2 is a depiction of a display showing various features of anembodiment of the present disclosure.

FIG. 3A illustrates an image and annotations, according to an embodimentof the present disclosure.

FIG. 3B illustrates non-embedded nature of the annotations in FIG. 3A,according to an embodiment of the present disclosure.

FIG. 4 depicts a map illustrating various stages of a treatment process.

FIG. 5 is a flowchart depicting another method, in accordance with anembodiment of the present disclosure.

FIG. 6 depicts a system, according to an embodiment of the presentdisclosure.

FIGS. 7A-7G depict example input data, in accordance with an embodimentof the present disclosure.

FIG. 8 is a screenshot including a plurality of images, in accordancewith an embodiment of the present disclosure.

FIG. 9 depicts a plurality of images including annotations, according toan embodiment of the present disclosure.

FIG. 10 depicts a plurality of images including annotations, accordingto an embodiment of the present disclosure.

FIG. 11 depicts an example output, in accordance with an embodiment ofthe present disclosure.

FIG. 12 depicts an example output table, according to an embodiment ofthe present disclosure

FIG. 13 depicts a system, in accordance with an embodiment of thepresent disclosure.

FIG. 14 is a flowchart depicting another method, according to anembodiment of the present disclosure.

FIG. 15 depicts a system including an application program, in accordancewith an embodiment of the present disclosure.

DETAILED DESCRIPTION

Referring in general to the accompanying drawings, various embodimentsof the present disclosure are illustrated to show the structure andmethods for image annotation systems and devices. Common elements of theillustrated embodiments are designated with like numerals. It should beunderstood that the figures presented are not meant to be illustrativeof actual views of any particular portion of the actual devicestructure, but are merely schematic representations which are employedto more clearly and fully depict embodiments of the disclosure.

The following provides a more detailed description of the presentdisclosure and various representative embodiments thereof. In thisdescription, functions may be shown in block diagram form in order notto obscure the present disclosure in unnecessary detail. Additionally,block definitions and partitioning of logic between various blocks isexemplary of a specific implementation. It will be readily apparent toone of ordinary skill in the art that the present disclosure may bepracticed by numerous other partitioning solutions. For the most part,details concerning timing considerations and the like have been omittedwhere such details are not necessary to obtain a complete understandingof the present disclosure and are within the abilities of persons ofordinary skill in the relevant art.

FIG. 1 illustrates a method for annotating a digital image with vectorannotations. As illustrated, method 100 includes receiving an image(depicted by act 102) to annotate. The image may include a raster basedimage (e.g., a bitmap image) and may be an image stored in one of manyavailable formats such as, for example, a JPEG, a BMP, a PNM, a PNG, aTIFF, and a PPM. The image, which may be retrieved from a storagemedium, may include a scanned image, a digital photograph, a workcreated on a computer (e.g., such as an architectural drawing), computedtomography, magnetic resonance image or any other valid source for adigital image. The image can be in a two dimensional or a threedimensional format.

Method 112 may further include annotating the image (depicted by act104). As will be appreciated, an annotation may include, for example, aregion of interest, a pointer, textual information, or any combinationthereof. A visible portion of the annotation may include, for example,the region of interest, the pointer, and the symbol. For example, in themedical field, a region of interest could be a feature or structure onan image (e.g., pathology, tumor, nerve) that conveys a clinical orresearch finding. Typically, a point, line, or polygon may be generatedto indicate a region of interest. A pointer may be defined by an authorand may be partially based on a computer depending on where the authorinitially places the pointer. For example, the author may select where atail of the pointer should be positioned, and an algorithm may calculatethe closest point on the region of interest to place a tip of thepointer. This method may enable the author to determine the layout ofvisual information on the image, without relying entirely automatedprocess, which may be unpredictable.

Textual information, which may include, for example, text, a symbol, alabel, a caption, or any combination thereof, may enable the author toadd knowledge on contents of an image. As will be appreciated, a symbolmay be drawn on an image without obscuring visual information orinterfering with other annotations. Further, a symbol may be used tolink visual annotation to textual information. In one example, a symbolmay comprise a lexicon specific piece of textual information enablingthe annotation to be linked to a larger body of information (e.g.,outside the image). A label may include word or phrase that defines thevisual annotation. A label may be derived from a lexicon or vocabulary,enabling dictionary-style lookup (e.g., via software). Lexicon-specifictextual information and/or symbols may be linked to a larger body ofinformation outside the image. A caption, which may include, one or morewords, may describe the annotation.

Throughout an annotation process, one or more authors may determinepresentation attributes, which define how annotations may be drawn whenrendered. Visible parts of presentation attributes (e.g., font, size,color, pointer type, and tip location) may also be interpreteddifferently depending on the medium (e.g. laser printer, journal articleor web browser). Further, presentation attributes may include numerousoptions to provide enhanced control over a presentation and annotationreuse.

For example, annotation size attributes may include “small,” “default”and “large.” This option may control a size of a pointer and associatedtext rendered with a visual annotation. Annotation color attributes mayinclude, for example, “light,” “default” and “dark.” As will beunderstood, this option may control the color of a region of interest(e.g., a polygon), a pointer, and any text, which may be rendered aspart of an annotation. Pointer type attributes may include, for example,“spot,” “line,” “pin,” “arrow” and “arrowhead.” Details (e.g.,appearance of pointers) may be controlled via a style sheet andrendering software. Further, pointer tip attributes, which may comprise,for example, “center” and “edge,” may control where a tip of a pointerappears (i.e., relative to the region of interest). Using the pointertip attributes, an actual pixel location of a pointer tip may bedetermined. Other embodiments may utilize free form placement of apointer and/or a pointer tip.

Method 100 further includes saving one or more annotations (depicted byact 106). The one or more annotations may be saved as vectorinformation, which is linked to the image. The vector information may besaved inside the image file or as a separate file. As will beappreciated, some image formats (e.g., PNG) allow the vector informationto be saved inside of the image file. Saving annotations, and possiblymetadata, as vector information may allow the annotations to be re-drawnor scaled and, thus, the presentation may be improved. In addition, theannotations may be modified for various reasons, such as, for example,to create interactive presentations.

As will be appreciated, annotations and metadata may be saved inside animage file with actual image information. Metadata may include anyinformation about the image or annotations, such as author name,including the date and time that the annotations were performed, thenames of persons who have viewed the image or annotations, including andthe date and time of the viewing. In a specific example related tomedical images, metadata may include patient information.

An annotation may also be stored in a separate file (i.e., apart from,but related to, the image). The text information may or may not beautomatically displayed (i.e., may or may not be visually hidden as adefault by an image viewer). It is noted that the text information maybe accessible for publishing, cataloging, displaying (e.g., interactivedisplay), and annotation drawing. As will be appreciated, storing textinformation (e.g., metadata and vector-based annotations) inside theimage file, may link the text information with the image information.

It is noted that the vector information can be stored in any format,such as in, for example only, extensible Markup Language (“XML”) format.It will be understood, storing the vector information in the XML formatmay ensure that annotations remain accessible as vector data (i.e., notembedded in an image), allow the annotations and images to becomere-usable, and maintain linkage between the annotations and images.Further, management of multiple versions in a proprietary format ordistribution of multiple copies of the same image may not be necessary.It is further noted that an output is not platform specific. In oneexample, a Scalable Vector Graphics (“SVG”) output format, which is anextension of the XML specification, may be used. Using SVG may providefor reuse, interactivity, and extensibility, interactive web viewing,and reuse. In addition, using SVG may enable the annotations and visualexpert knowledge (i.e., labels and captions) to remain linked to theimage.

Method 100 further includes rendering the image with the one or moreannotations (e.g., for display via an electronic display device)(depicted by act 108). It is noted that embodiments of the presentdisclosure may be configured for cross-media distribution. Statedanother way, embodiments described herein may enable annotated images tobe converted from one form of media to another form of media.

With reference to FIG. 2, a display 150 including an image 152 andannotations is depicted. More specifically, display 150 includes severalregions of interest associated with image 152. One region of interest,as indicated by reference numeral 154, is noted by a label 156 (i.e.,“Cyst”), which is connected to region of interest 154 by a pointer 160.In addition, display 150 includes a caption 162 and a symbol 164 forregion of interest 154. Display 150 also includes label (i.e., “CarotidArtery”) connected to the depicted carotid artery via pointer 168, alabel (i.e., “Vagus Nerve”) connected to the depicted vagus nerve via apointer 172, and a label (i.e., “Carotic Sheath”) connected to thedepicted carotic sheath via a pointer 176. As will be appreciated, theannotations are useful in conveying information to an observer.

In accordance with an embodiment of the present disclosure, a separateannotation file may contain a digital signature of the image file incase the two files are separated. As will be explained in greater detailbelow, reuse of the image is facilitated since the original imageremains unchanged and the annotations remain linked to the image. Itwill be appreciated that because the annotations are not embedded intothe image, they can be referenced, grouped (as shown in FIG. 2) andindexed for a variety of purposes. In addition, while multipleannotations can be added to an image, not all of the annotations need bedisplayed at the option of the presenter, to create a contextappropriate annotated image. These multiple annotations can beinteractive.

FIGS. 3A and 3B respectively depict a display including an image and adisplay including the annotations without the image. FIG. 3A illustratesmarked regions of interest with respective pointers and labels. As willbe appreciated, the annotations may be “overlaid” over the originalimage and not embedded. Rather, the annotations may be stored in aseparate file, which may be linked to the image file. In one specificexample, the annotations may be stored in an image independent vectorformat (i.e., for high-resolution display at all scales). It is notedthat the image is unedited and no pixels of the original raster imagewere modified.

FIG. 4 depicts a cognitive map 200, which illustrates the complex natureand stages of a medical treatment process. As will be understood by aperson having ordinary skill in the art, a treatment process may involvevarious stages and interaction amongst various healthcare personnel frommultiple healthcare specialties. Further, the treatment process mayinclude interaction with images, patient information, payers, and apatient.

As will be appreciated, diagnostics and treatment planning may bedependent on bringing members of a care team together for routine reviewand updates of a treatment plan. For example, in a breast cancer case(e.g., see FIG. 4), data (e.g., imaging and textual inputs) fromdifferent specialties (e.g., radiology, surgery, pathology, etc.)perform a critical role in diagnosis. The data may originate fromdifferent encounters generated at different times during treatment of apatient. Treatment requires ongoing collaboration between health careprofessionals (e.g. expert clinicians) of different specialties thatform a multi-disciplinary team, which requires access to the patient'sdata for critical decision making. Ideally, each healthcare professionalshould review all the patient's data (e.g., within a tumor board or casereview) including image data and annotated areas of concern for aparticular patient. Further, text records that provide context for theimage review such as radiology, surgical and previous pathology reports,remain vital.

A radiology-pathology-oncology breast cancer example is a commonscenario. In this example, a female may receive a mammogram that shows asuspicious mass. The mass may be biopsied and sent to pathology where aninitial diagnosis begins with screening and identifying suspiciousfindings from imaging (e.g., via radiology). Further, tissue diagnosis(e.g., via pathology) imaging may occur. Other types of pathologyimaging and oncology imaging are routinely accessed and reviewed inmulti-disciplinary team evaluation. Moreover, it is becoming more commonthat additional studies, using biomarkers and sophisticated molecularimaging, may provide the framework for pathologists to contributesignificantly to refine treatment pathways that will become routine inpersonalized therapy. In the breast cancer example (e.g., wherein apatient is diagnosed with breast cancer), images and data critical tothe treatment plan may be generated over an extended time period (e.g.,12-18 months).

Record systems, such as, for example only, electronic medical/healthrecord (EMR/EHR) systems, may be designed to manage and depictinformation. For example, EMR/EHR systems may be configured to manageand depict administrative, logistical and systematic health informationfor a patient. Current EMR/EHR systems maintain a rigid view of apatient care process and struggle to integrate disparate informationsources and heterogeneous provider systems. EMR systems typically do notmeet the requirements of a fully engaged multi-disciplinary teamprocess, especially when members of a care team are from disparateorganizations. Further, conventional EMR systems may fail to integratemulti-modal imagery, complete annotation, and incorporatemulti-disciplinary team contributions to diagnosis, treatment planning,and treatment plan execution. These ridged processes are typically basedon a financial model that consistently adds cost to a care processwithout improving outcomes and/or patient satisfaction.

Further, in the medical field, for example, text-based patientinformation (e.g., findings, diagnoses and clinical summaries) mayreside in separate information systems, which may prohibit timely accessby a care team. For example, clinical notes, which may reside in anEMR/EHR and a laboratory information system (LIS), may not be connectedto each other or to the imaging systems.

In addition, some conventional systems, which may manage patient recordsin a chronological or linear manner, are focused on patient records andare not designed to facilitate or support any level of interactivity(e.g., collaboration including specialized tools for visual annotationand real-time communication). These systems may consume images, providedeach image is a manageable size, but without the contextual componentsabout the image or set of images leaving the image as an outlier in thesystem.

Various embodiments of the present disclosure relate to a system (e.g.,for clinicians, such as radiologists, pathologists, oncologists,gastroenterologists, and primary care physicians) configured to supportdiagnosis and treatment planning (e.g., in a virtual tumor board or casereview environment). The system may be configured for exchange andmanagement of data including, for example, images and text. It is notedthat data (e.g., images and/or text) having a common identifier may beincluded within a data set. Stated another way, all data within a dataset may have a common identifier, such as a patient ID. In one or moreembodiments, the system may include a collection of technologies (e.g.,modules and/or components) configured to exchange and manage data (e.g.,images and/or text) within a data set.

Compared to conventional systems, which may receive images from a singlemodality (e.g., radiology), various systems, as described herein, areconfigured to receive, process, and aggregate data (e.g., text and/orimages) from different modalities (e.g., radiology, pathology, oncology,dermatology, GI, etc.) into a data set. Thus, a data set may includedata from different image modalities and/or medical specialties tosupport and facilitate evaluation (e.g., multi-disciplinary teamevaluation), review, and treatment planning (e.g., collaboration).According to various embodiments, relationships among data within a dataset may be maintained. For example, data within a data set may be linkedvia metadata.

A system may be configured to provide a presentation layer (e.g., via acomputer screen) or portal that allows for real-time communication withdata (e.g., images), supporting the iterative nature of amulti-disciplinary team by allowing specialists to access and reviewcritical information, including clinical summaries from otherspecialists. Bringing specialists and providers together to evaluateimaging and diagnostic summaries for a case, in stages, in a timelymanner, may improve the treatment outcome of the patient. For example, apathologist may consult with a surgeon and an oncologist. At every stepin a treatment process, images may be generated from multiple devicesthat are unlinked to the patient's health record.

Diagnosis and findings may be important for devising a treatment planand may require interaction across various specialties (e.g.,gastroenterology, radiology, surgery, pathology, oncology, pulmonology,primary care, etc.) to identify the correct treatment planning options.Once treatment is underway, studies may be conducted over a time period(e.g., several months) to determine the effectiveness of treatment andto determine whether changes are necessary. Studies (e.g., imagingstudies) may be required to measure the effectiveness of treatment andgenerate data (e.g., textual findings and images). This data may bereviewed by a care team (e.g., radiology, pathology, and oncology) informal reviews (e.g., case reviews or tumor boards). Further, additionaldata (e.g., comments and findings) may be collected regarding the statusof treatment and about best practices for refining treatment.

In one specific embodiment, a system may process and aggregate imagesincluding annotations (e.g., annotated regions of interest) across imagespecialties and modalities (e.g., radiology, pathology, surgery andoncology), wherein the annotations (e.g., metadata) may maintainrelationships across images one or more images within a data set.Annotated regions of interest (ROI) may maintain a relationship with anoriginal image that may be part of a data set for a given record orcase. Annotated ROI may include annotated information to describe arelationship between and across images from a different modality ormodality as a group to convey meaning in diagnosis or treatment. Theinformation may be modified (e.g., across clinical imaging modalities)to reflect a change in a record for a given diagnosis and/or treatmentplan.

A system may further be configured to group one or more annotations intodiscrete elements, wherein the one or more annotations may be displayedwith one or more images to maintain the data-derived parent-childrelationships across different images and textual data from differentdata modalities. The one or more annotations may provide context to oneor more images and/or text in human-readable and machine-consumableformats. As will be appreciated, in various industries (e.g.,healthcare, geography, oil and gas, and satellite industries),annotations, and the ability to produce a context specific collection ofannotation groups and their metadata layers may be useful for conveyingmeaningful information for all aspects of digital imaging. It is notedthat although various embodiments of the present disclosure aredescribed with reference to healthcare, the present disclosure is not solimited. Rather, other applications, including, for example, geographyapplications, oil and gas applications, and satellite applications, arecontemplated within the scope of the present disclosure. Further,embodiments of the present disclosure are not limited to digital data.

In another specific embodiment, a system may be configured to aggregate,organize and display field of view (FOV) images, wherein the FOV imagescan be modified to reflect a change in treatment plan or diagnosis.Further, in one embodiment, annotated regions of interest or one or moreFOV images may be included in a standardized format (e.g., theContinuity of Care (CCD) reporting format), which may be compliant withthe Clinical Document Architecture (CDA). More specifically, in anotherembodiment, a system may be configured to aggregate and organizespecific images (e.g., FOV images) and their annotated regions ofinterest for inclusion into structured output (e.g., the CCD structuredoutput (i.e., a standard format)) while maintaining relationships withthe data (e.g., images and associated textual data) with externalsystems via metadata layers of the annotations. Further, a system maygenerate (e.g., output) a combination of annotation regions of interestcaptured in a FOV from a combination of encounters (e.g., for thepurpose of conveying specific information regarding a diagnosis orchange in diagnosis).

In one example, a treating oncologist may be able to review with themulti-disciplinary team care team, and with the patient, the annotatedregions of interest within different images associated with thediagnosis and generate a CCD integrated output, which can be used forhuman or system communication such as and electronic medical record(EMR) or electronic health record (EHR).

In one embodiment, the format and type of an output of a system may be acombination of annotation regions of interest captured in a FOV from acombination of encounters (e.g., for the purpose of conveying specificdata regarding a diagnosis and/or documentation of relevant changes to adiagnosis). Annotated regions of interest contained in the FOV imagesmay contain relevant data, such as billing codes, lexicons, andvocabularies for external system reference and consumption. The systemmay index and catalogue all received data for the purpose of search andretrieval across data sets. For example, a number of patients that havereceived a specific treatment and are in remission may be determined(e.g., via a search).

Moreover, a system may extract specific FOV annotated images forinclusion into a structured output. Also, FOV images may be capturedwhile maintaining annotated regions of interest as a metadata layer, andthe FOV annotated images may be provided in a structured output into aCCD structure, which is compliant with the CDA. Furthermore, annotationsand other information (e.g., information contained in an annotation,such as in a label and/or caption) may be stored with an image or insideof an image, and may be exported to, for example, theConsolidated-Clinical Document Architecture (C-CDA) standard forstructured reporting. FOV images, including annotated regions ofinterest (e.g., visual, label, caption), may be collected for inclusioninto the C-CDA structured output for consumption by an external system,and metadata (e.g., annotations) may include information, which may beprocessed by the external system. Further, a system may be configured tocapture FOV images while maintaining annotated regions of interest(e.g., as a metadata layer) in a manner that is compliant with theHealth Information Technology Standards Panel (HITSP) standard, whichmay allow for external consumption by a separate system.

In one embodiment, annotations (e.g., labels and captions) can beextended to contain discrete pieces of information for clinicaldocumentation, billing and reimbursement, and can be used or reused byan external system to communicate with other external systems. Variousembodiments may include tools for visually annotating images with, forexample, symbols, labels, captions, billing codes, and vocabularies.These tools may allow for tracking events (e.g., during treatmentplanning), and, therefore, may allow information to be added as part ofa documentation process. Documenting different and iterative cycles andmaintaining a collection of information and knowledge generated by acare team becomes part of the encounter-based reporting and potentiallybilling. As an example, the addition of a semantic ontology may provideflexibility for indexing and decision support extending the value of theinformation in the collection. Analytics companies can easily integratewith the system and use the collection of data for a variety of uses(e.g., clinical decision support, patient trending, and reimbursementtrends).

Various embodiments, as disclosed herein, may be compliant with standardHealth Level 7 (HL7) protocols for interfacing to external systems,Integrating the Healthcare Enterprise (IHE) profiles, as well asstructured (XML, SVG) messaging for CCD transactions. Moreover,embodiments may be configured to adhere to standard security practices(e.g., maintain Health Insurance Portability and Accountability Act(HIPAA compliance) with user authentication in a role-based accesscontrol environment.

FIG. 5 is a flowchart of a method 300, in accordance with an embodimentof the present disclosure. Method 300 includes accessing one or moreimages (depicted by act 302). Further, method 300 includes defining oneor more regions of interest in the one or more images while maintainingrelationships for the one or more images (depicted by act 304). Morespecifically, in one example, a region of interest may be defined in atleast one image of a data set, and a relationship of the region ofinterest may be maintained across two or more images in the data set.Further, method 300 may include generating one or more annotations(e.g., desired symbols, labels and pointers) in the one or more imageswhile maintaining relationships for the one or more images (depicted byact 306). More specifically, for example, at least one annotation for atleast one image in a data set may be generated, and a relationship ofthe annotation may be maintained across two or more images in the dataset. Further, method 300 may include determining whether additionalannotations should be added (depicted by act 308). It is noted that aperson may add annotations to an image, which was already annotated byanother person (i.e., “multi-user authoring”). As an example, more thanone doctor may annotate the same medical image.

Method 300 may further include grouping the one or more annotationsacross two or more data sources (e.g., imaging sources) (depicted by act310). Method 300 further includes organizing the one or more regions ofinterest in the one or more images (e.g., into context-appropriateviews) (depicted by act 312). Context-appropriate viewing (i.e. of animage and related annotations) allows the annotations to be turned on oroff for a particular audience. The annotations may be separate from theimage, and may be separate from each other, thus, depending on thecontext, portions of annotations may be viewed in a presentation whileother portions remain hidden. Stated another way, a view attribute canturn annotations on/off in a context-appropriate manner. Method 300 mayalso include saving the file with annotations as vector informationlinked to the image (depicted by act 314)

Further, method 300 may include capturing FOV images while maintainingdata relationships (depicted by act 316). For example, relationships ofannotated regions of interest may be maintained across image modalitiesas a metadata layer.

FIG. 6 depicts a system 400, according to an embodiment of the presentdisclosure. System 400 includes a processor 402, an aggregator 404, andorganizing module 406, an organizing module 408, an annotation servicemodule 410, and a template module 412. System 400 may be configured toreceive data inputs 414, which may comprise images (e.g., from clinicalimaging devices), textual information (e.g., from clinical systems), ora combination thereof. Further, system 400 may be configured to processand aggregate data inputs 414, and generate one or more outputs 416,which may comprise machine-readable data (e.g., to be conveyed to anexternal system), human-readable data (e.g., context appropriate view orpresentation) or a combination thereof.

Data inputs 414 may be associated with an identifier (or nativemetadata), and data inputs 414 having a common identifier may beincluded within a data set. Stated another way, for example only, allreceived data associated with a specific patient may have a commonidentifier and, thus, may be associated with a common data set. Further,data in a data set may originate from, for example, different imagemodalities and medical specialties.

By way of example, data inputs 414 may comprise data, such as textualdata (e.g., structure or unstructured textual data), image data (e.g.,structure or unstructured image data), ancillary data (e.g., clinicaldata, notes, references, etc.), or any combination thereof. Data inputs414 (e.g., text, images, etc.) of system 400 may be generated viavarious scenarios. For example, various medical diagnosis and/ortreatment processes (e.g., related to breast cancer, lung cancer, skincancer, colon cancer, etc.) may generate data inputs 414.

Some example scenarios for generating data inputs 414 will now bedescribed. By way of example, diagnosing breast cancer begins with ascreening mammogram that generates a set of findings (e.g., text andimages) that identify a suspicious mass or lump in the breast tissue. Afurther imaging study, usually a breast magnetic resonance imaging (MRI)and ultrasound may be required to examine the suspect tissue anddetermine next steps. Once the suspect mass is identified an ultrasoundguided needle core biopsy may collect a piece of tissue from theaffected area. The tissue sample may be sent to the pathology lab for adiagnosis. Depending on the diagnosis, the patient either receives adiagnosis of benign and is continually monitored or scheduled for atumor resection in the event the needle core biopsy is malignant. If thediagnosis is malignant, the patient may be scheduled for a tumorresection and further histological interpretation to determine the typeof breast cancer. The pathologist may receive the tissue from the tumorresection and may conduct different tests and studies to determine thetype and severity of the breast cancer. Each test and study may generateseveral images from different devices and text from different systems.The diagnosis of breast cancer may be dependent on the availability ofhaving data (e.g., images and text) from, for example, radiology,surgery, and pathology to determine the best treatment plan options forany given patient. In the case of breast cancer, the diagnosis andfindings are important for devising the treatment plan and require asummary and communication across the specialties (e.g., radiology,surgery, pathology) to identify the correct treatment planning options.

Further, lung cancer may be diagnosed upon a patient exhibiting avariety of lung related symptoms. More specifically, for example, apatient, exhibiting a variety of symptoms, such as chronic coughing,chest pain, blood with cough and other symptoms, may be seen by aphysician (e.g., primary care physician). The physician may order aspecialty set of pulmonary tests combined with other tests such as CT orMRI, as well and needle core or ultrasound guided biopsy to determinethe type and severity of lung cancer. The pulmonologist, radiologist andpathologist may work together to review inputs from each specialty toarrive at a diagnosis and continue to review the inputs as the patientundergoes treatment planning. Depending on the type and severity of thecancer, radiation therapy may generate additional images and textualinformation from other affected parts of the body to measure how thepatient is responding to treatment. Images and textual information maybe generated from multiple imaging modalities and specialties across thecare landscape that are reviewed by all specialty and sub-specialists inthe treatment of the patient.

Skin cancer (e.g., melanoma) is an aggressive cancer. When detected,skin cancer may require one or more physicians on a care team to haveaccess to images and/or text (e.g., data inputs 414) that are generatedat various stages in the patient's treatment plan. The patient may beseen by, for example, a family physician, or at an emergency room if inlate stage melanoma, with symptoms ranging from chest pain, stomachpain, dizziness and fatigue. The patient may undergo imaging, CT, MRI orboth, to allow physicians (e.g., radiologists) to view affected areas ofthe anatomy. The patient may undergo a biopsy where the tissue is seenby another physician (e.g., pathologist) to render a diagnosis. Thepathologist and an oncologist may review treatment planning options fora given patient and determine what plan to initiate that provides themost effective treatment and results. The oncologist may use imagingfrom PET/CT to review how well the patient responds to treatment. Allstages in a treatment process may generate data (e.g., multiple imagesand textual information) inputs that are necessary to measure how well apatient responds, justify treatment, and monitor patients throughout theprocess.

Diagnosing colon cancer may begin with a colonoscopy, which may generatedata (e.g., text and images) that may identify polyps or lumps in thecolon, duodenum or stomach. Polyps may be removed and polyp tissue maybe sent to the pathology laboratory to determine a diagnosis. Dependingon a diagnosis, the patient may either receives a diagnosis of benignand is continually monitored or scheduled for further imaging studies todetermine the type and severity if malignant. A further imaging study,usually a colorectal ultrasound, Computed Tomography (CT) and/or MRI maybe required to look closely at the suspected area for diagnosticevaluation. A pathologist may receive tissue from the tumor resectionand conduct different tests and studies to determine a type and severityof colon cancer. Each test and study may generate several images fromdifferent devices and text from different systems (e.g., pathology,oncology, etc.). The diagnosis of colon cancer, like all cancers, maydependent on the availability of having the previous data (e.g., imagesand text) from, for example, gastroenterology, radiology, surgery, andpathology and oncology to determine the best treatment plan options forany given patient.

Is it noted that the data inputs (e.g., images and textual information)received by system 600 may be unrelated and often is not generated fromthe same system or from the same institution. For example, the datainputs may originate from geographically separate areas (e.g.,geographically isolated clinics, laboratories and hospitals). Further,the data inputs may be generated over long periods of time for a givenpatient record.

FIGS. 7A-7G depict example inputs (e.g., inputs 414), which may bereceived by system 400 (see FIG. 6). Specifically, FIG. 7A depictsOphthalmic image of the retina showing onset of retinitis pigmentosa andFIG. 7B depicts Pathology Fluorescence in Situ Hybridization (FiSH)image showing biomarkers on cancer cells. FIG. 7C illustrates aPathology Hematoxylin and Eosin stained tissue sample, and FIG. 7D is aPathology Graph output from Kristen Rat Sarcoma oncogene (KRAS) showinghigh levels of oncogene protein. FIG. 7E depicts Pathology Beta-typePlatelet-derived Growth Factor (PDGFRB) showing prolific celldifferentiation in cancer, and FIG. 7F Gastroenterology Colon polypimage during routine colonoscopy. Further, FIG. 7G is a RadiologyTransverse Computed Tomography (CT) of the lung and chest.

Referring again to FIG. 6, processor 402 is configured to receive one ormore data inputs 414. Further, processor 402 may identify a relationshipamongst data inputs (e.g., identifying data inputs that share a commonidentifier). In one embodiment, processor 402 may receive data inputsand associated metadata at a gross level. Processor 402 may collectivelyact as a data exchange and accept data inputs (e.g., images and text),which are received (e.g., incrementally) by system 400. In one example,processor 402 may receive data inputs under a source schema and process(e.g., restructure or transform) the data input into a target schema sothat the processed data is an accurate representation of the source data(i.e., data inputs 414). Processor 402 may normalize the data inputs tobe received by aggregator 404. Further, processor 402 may communicatewith aggregator 404 to manage data so that data is maintained together(e.g., in a data set).

Aggregator 404 is configured to receive processed data (e.g., fromprocessor 402) as well as associated metadata. Aggregator 404 may ensureall processed data from processor 402 is managed such that no data islost. Further, aggregator 404 may manage and assign the processed datato a record ID, which may be maintained by aggregator 404. In oneexample, aggregator 404 not only manages the gross level metadata butmay also assign and/or generate additional metadata across data sets(e.g., as a collection for presentation to a user interface and/or someother output (CCD, report, etc.).

Aggregator 404 may be configured to collect the data sets for, forexample, presentation to a user interface. Further, aggregator 404 maycall annotation service module 410 to present annotationtools/functionality (arrows, labels, captions, etc.) together with thedata set to a user interface. A user interface (e.g., as managed byaggregator 404) may allow for a clinicians, or any other users, to viewany or all parts of the data sets received by aggregator 404. Further,the user interface may allow a user to annotate at least a portion ofthe data within one or more data sets. Users can add descriptions to oneor more images that have meaning or describe a specific regions ofinterest that has a relationship to another region of interest in one ormore other images of a data set. Different users may (e.g., viaannotation service module 410) add iterative annotations, labels and/orcaptions that have additional meaning to any combination of the imagesand text, which may be part of one or more data sets. In one embodiment,annotated information may be managed by aggregator 404.

Aggregator 404 may also aggregate data that may be packaged intoincrements for a report that is consumed by an external system, a humanreadable report through template module 412, or annotation servicemodule 410. Aggregator 404 may also be configured to consolidate andmanipulate data sets via metadata.

Aggregator 404 may be configured to interact with annotation servicemodule 410 and organizing modules 406 and 408. For example, aggregator404 may be configured to interact with annotation service module 410 andorganizing modules 406 and 408 to maintain and manage the annotationswith images and collect and combine the data that is requested byorganizing modules 406 and 408.

In one embodiment, aggregator 404 may include a semantic layer thatmaintains the meanings across the different data inputs (i.e., receivedby processor 402) as well as meanings that are added through annotationservice module 410. For example, a meaning can be derived from anannotation, label, caption, etc. that contains a description of abnormalanatomy added by a physician. This description may be added to data thatwas consumed from the processor on a prior data (e.g., Jan. 18, 2015).Further, when comparing the data from the prior date (e.g., Jan. 18,2015) with data that is presented on a subsequent date (e.g., Mar. 8,2015), a clinician (e.g., a physician), or any number of clinicians, mayadd annotations, labels, captions, etc. that may contain a differentwork or phrase but has meaning related to the annotation from the priordate (e.g., Jan. 18, 2015).

Aggregator 404 may manage the meaning (e.g., the information added initerative cycles, as described the example described above) as anadditional layer of metadata across the data set to the record ID for agiven patient or instance (e.g., medical, oil and gas, geography, etc.)through the semantic layer. It is noted that a semantic layer mayinclude annotations (e.g., arrows, pointers, circles, and lines),labels, captions and narratives that have meaning for any given user,and which may be linked to a syntax that links the meaning across thedata. Aggregator 404 may be configured to maintain relationshipsbetween, for example, words, narrative, labels and symbols.

In one embodiment, aggregator 404 may be configured to manage andmaintain the relationships of the data inputs (i.e., within a data set)through the use of a semantic ontological framework. The ontologicalframework (e.g., specific to medicine) may be configured to extract andaggregate information collected from, for example, iterative cycles, asdescribed above. The ontological framework, which may use a commonvocabulary in which shared information or knowledge is represented, mayalso prepare collections of information to allow a user to add adifferent code (e.g., CPT billing code) to the annotation set. In oneexample, the annotation set may be submitted to a billing service wherethe collection of annotated images with their respective regions ofinterest, labels and captions, along with the billing code, may becollected and queued up for organizer module 406 to send to templatemodule 412 for output.

A semantic-ontological framework may be internal to aggregator 404, andmay manage the meaning, which may be part of the collection of largerdata sets, and along with other functions of aggregator 404, may be usedto communicate with organizing module 406. By way of example, aggregator404 may generate an output that represents one of the iterative cycles,as described above, with a specific meaning (e.g., the tumor isshrinking) that is documented by, for example, a team of clinicalspecialists. Further, the output may include a billing code that alsoincludes a quality measure, which may be linked to the data set for thatinstance at that time. The output may be consumed by organizer module406 and sent to template module 412. This is one example scenario, andother scenarios, in which aggregator 404 manages a data set, may exist.

Organizing modules 406 and 408 may each be configured to receive inputs(e.g., images and/or text) from both aggregator 404 and annotationservice module 410. Organizing modules 406 and 408 may capture inputswith meaningful information contained as, for example, text,annotations, labels and captions for output to template service module412 or to an external system. Further, organizing modules 406/408 may beconfigured to maintain relationships across aggregated data andannotations that are added in a temporal sequence. Organizing modules406 and 408 may be configured to capture a selection of inputs for agiven time sequence including FOV information and send that to templatemodule 412 for output to an external system.

Annotation service module 410 may be configured to annotate both imagesand text, which have been processed and aggregated, and is furtherconfigured to maintain relationships across the collection of data. Inaddition, annotation service module 410 may be configured to annotateregions of interest on a single image, annotate regions of interestacross images in a data set (e.g., such that the annotations maintain aspecific relationship for a given point in time), store data (e.g.,codes and lexicons) for the purpose of documentation, capture FOV thatrepresent an ROI that are sent to the template, or any combinationthereof. Annotation service module 410 may provide a user with theability to interact with images and also maintains relationships throughthe annotations, labels and captions across the data set.

Template service module 412 is configured to properly format the outputfrom the processed inputs, along with the annotations and relationshipswith the textual data into a template, which may be in compliance with astandard machine readable output (in the case we are adhering to theClinical Document Architecture (CDA) Continuity of Care (CCD) standardfor external data exchange.

Various embodiments as disclosed herein relate to analyzing andcollecting information to develop high-specificity libraries for spatialmapping of cellular/molecular profiles combined with clinical expertiseand user-defined markup throughout the multi-disciplinary team andthrough annotation service module 410. Health care professionals (e.g.,clinicians) from multiple specialties may view disparate images from,for example, radiology to complex pathology, in a single view, whereinthe cell profiles are linked and mapped across the images for a singlecase. Further, annotations can identify molecular signals that can bequantitatively mapped across other radiology, pathology, oncology,surgery images, and cell types. Cell class data obtained from cellanalysis and mapping may be combined with end-user collaborationmetadata in a structured format for use in, for example, cellphenotyping, treatment planning and disease profiling.

FIG. 8 is a screenshot (e.g., a presentation to a user interface)depicting a plurality of images, such as an ultrasound image 502, aprogesterone receptor (PR) image 504, and a breast magnetic resonanceimaging (MRI) image 506. FIG. 8 further includes a human epidermalgrowth factor receptor (HER2) image 512 depicting tissue biopsiespresented together with breast MRI images 506 and 508. FIG. 8 furtherincludes a positron emission tomography and computation tomography(PET/CT) 510, which may be used to determine the anatomic area andexpression of genes that have an important role in the development andprogression of certain aggressive types of breast cancer. As will beappreciated, FIG. 8 depicts annotations, such as lines, arrows, andcodes, that may be maintained across at least some of the images, andmay be used for, for example, document treatment planning, billing,and/or patient outcomes.

FIG. 9 depicts aggregated data including breast MRI 608 and 610, andcolocalized PET/CT 612 linked to the tissue biopsies and stained ER 602,PR 604, and HER2 606. For example, one or more of the images depicted inFIG. 9 may be used in determining how much overexpressed HER2 protein ispresent and binding reactions or signals in a patient with triplenegative breast carcinoma. The annotated regions of interest 605A and605B (e.g., a line and an arrow) may link the specific regions ofinterest for each type of input and may provide the ability to compare,for example, the H&E and HER2 tissue biopsies. As will be appreciated,FIG. 9 depicts annotations 605 (e.g., lines, arrows, text, labels,circles and codes) that may be maintained across at least some of theimages, and may be used for, for example, document treatment planning,billing, and/or patient outcomes.

FIG. 10 depicts a plurality of images 702, 704, 706, and 708 fromvarious modalities (i.e., H&E, Immunohistochemistry and FISH). FIG. 10further depicts annotations 705A-705D. FIG. 10 illustrates an example ofhow cells identified in a FISH image may be mapped to H&E and HER2images of stained tissue sample that have an established relationshipfor a variety of outputs. As will be appreciated, FIG. 10 depictsannotations 705 (e.g., lines, arrows, text, circles, labels, and codes)that may be maintained across at least some of the images, and may beused for, for example, document treatment planning, billing, and/orpatient outcomes.

According to various embodiments, a system (e.g., system 400; see FIG.6) may enable an end-user to use annotation tools for drawing a regionof interest (ROI) on various images, such as image 504 (see FIG. 8) witha Common Procedural Terminology (CPT) code (e.g., 88369), wherein thecode may maintain a relationship to a given image and cohort images bysystem 400. More specifically, codes may be used to link data (e.g.,images) within a data set.

FIG. 11 is an example output 800 (e.g., output 416; see FIG. 6), whichincludes a CCD-A integrated report including annotated regions ofinterest. More specifically, FIG. 11 depicts an integrated report withannotated regions of interest for a breast cancer case. A CCD-Aintegrated report may be formatted such that it may render as a readableportable document or may be sent to a patient record in a structuredformat, which is consistent with the industry CCD-A communicationstandard. As illustrate, output 800 may include annotations.

Generating a report that encapsulates a collection of data (e.g.,diagnostic events, specific FOV from images with annotated regions ofinterest (e.g., that document the expert findings across themulti-modality care continuum)) may be important for medical treatment(e.g., cancer treatment). According to various embodiments, FOV imagesmay contain metadata that maintains relationships across a data set.

Some embodiments, as disclosed herein, relate generally to aggregating,organizing, and reporting in a standardized format and, morespecifically, to collecting field of view images with metadata andaggregating that information with other patient information forgenerating one or more outputs (e.g., reports). In one embodiment,various systems may be configured for generating an output for reportingto the Centers for Medicare and Medicaid Services (CMS) to meet therequirements for the Physician Quality Reporting System.

As will be appreciated by a person having ordinary skill in the art, CMShave several quality initiatives that provide information on the qualityof care across different settings, including hospitals, skilled nursingfacilities, home health agencies, and dialysis facilities for end-stagerenal disease. These quality initiatives may aim to empower providersand consumers with information that would support the overall deliveryand coordination of care, and ultimately may support new payment systemsthat rewards physicians for providing improved quality care, rather thansimply paying based on the volume of services.

Under the Tax Relief and Health Care Act of 2006 (TRHCA), CMSimplemented the Physician Quality Reporting Initiative (PQRI) (nowcalled Physician Quality Reporting System (PQRS)) with a bonus paymentof 1.5 percent for successful participation based on the estimated totalallowed charges for all cover services during the reporting period.Physicians and non-physician providers who participate in the programtransmit data to CMS regarding the quality measures reported on incaring for their Medicare patients. Under the Medicare Improvement forPatients and Providers Act of 2008 (MIPPA), the PQRS program was madepermanent. MIPPA also requires CMS to post on a website the names ofeligible professionals and group practices who have satisfactorilyreported under the PQRS. This information, along with additional measureperformance information, is now posted on the Medicare Physician Comparewebsite.

Several PQRS program changes were included in health care reformlegislation enacted in 2010. The Affordable Care Act (ACA) requires theimplementation of timely feedback and the establishment of an informalappeals process by year 2011. The ACA also calls for PQRS paymentpenalties starting in 2015. CMS finalized, in its 2012 MedicarePhysician Fee Schedule rule, that 2015 program penalties are based on2013 performance. Therefore, physicians who elect not to participate orare found unsuccessful during the 2013 program year, will receive a 1.5percent payment penalty, and 2 percent thereafter. In the 2014 MedicarePhysician Fee Schedule Rule, CMS finalized its proposal to base 2016PQRS penalties off of 2014 reporting. Therefore, physicians who did notparticipate in PQRS in 2014 will receive a 2 percent penalty in 2016.The year 2014 was the last year a physician could receive an additionalincentive for participating in the PQRS Maintenance of Certification(MOC) program or receive an incentive for participating in PQRS. In the2015 Medicare Physician Fee Schedule Rule, CMS finalized its proposal tobase 2017 PQRS penalties off of 2015 reporting.

PQRS is a reporting program that uses a combination of incentivepayments and negative payment adjustments to promote reporting ofquality information by eligible professionals (EPs). The PQRS programprovides an incentive payment to practices with EPs (identified onclaims by their individual National Provider Identifier (NPI) and TaxIdentification Number (TIN)). EPs satisfactorily report data on qualitymeasures for covered Physician Fee Schedule (PFS) services furnished toMedicare Part B Fee-for-Service (FFS) beneficiaries (including RailroadRetirement Board and Medicare Secondary Payer).

Beginning in 2015, the PQRS program also applies a negative paymentadjustment to EPs who do not satisfactorily report data on qualitymeasures for covered professional services. The Medicare PhysicianCompare website serves as the primary and authoritative source for allpublicly available information and CMS-supported educational andimplementation support materials for PQRS.

A qualified clinical data registry (QCDR) is a new reporting mechanismavailable for the PQRS beginning in 2014. A QCDR will complete thecollection and submission of PQRS quality measures data on behalf ofEPs. For 2014, a QCDR is a CMS-approved entity that collects medicaland/or clinical data for the purpose of patient and disease tracking tofoster improvement in the quality of care provided to patients. EPs whosatisfactorily participate in PQRS through a QCDR may earn the 2014incentive payment (0.5%) and avoid the 2016 payment adjustment (2.0%).To be considered a QCDR for purposes of PQRS, an entity mustself-nominate and successfully complete a qualification process.

Various embodiments of the present disclosure relate to collecting data(e.g., FOV images with metadata) and aggregating the data with otherpatient information for reporting to CMS to meet the requirements forthe QCDR that is PQRS. Various embodiments may be configured to collectand report on the image-based Quality Measures (QM) that are identifiedin the PQRS. Conventional solutions only address text-based QM and failto address the specialty specific QMs in pathology or cancerdiagnostics. In contrast, various embodiments disclosed herein providefor aggregation of data for reporting across unformatted systems thatfunction as a definitive source of treatment plans and treatmentsummaries. Further, various embodiments may provide both textual andimage based summaries (e.g., a CCD-A integrated report). As will beunderstood, QMs are designed to address and document effective clinicalcare, care coordination and cost reduction and efficiencies.

FIG. 12 includes a table 900 titled “Focus on Dermatopathology: SkinCancer Quality System.” As will be understood by a person havingordinary skill in the art, table 900 may represent a process of aquality system for dermatopathology specifically, melanoma. The QMs (inaddition to the Federal core measures) are PQRS 137, 138, 194 and 224.Each of these QMs depend on capturing a series of encounters anddocumenting them through an accounting system that also provides a meansfor communication with the primary care or dermatologist in theparticular case. To specify the methodology, embodiments of the presentdisclosure are extended to collect various encounters (regardless ofimage or multi-disciplinary team), consume information from outboundhealth level 7 (HL7) feed as an encounter, and index the collection ofdata linked to a master patient index (MPI). One deficiency ofconventional systems and methods is revenue accounting to triangulatethe information for accounting purposes and reporting.

An example of a single PQRS reporting will now be described. Hospitaladministrators and physicians may be able to perform certain nationalquality metrics through via one or more of the embodiments of thepresent disclosure. The first quality metric that may be available toall participating sites may be PQRS measure #147, which is titled“Nuclear Medicine: Correlation with Existing Imaging Studies for AllPatients Undergoing Bone Scintigraphy” and includes the followingdescription: Percentage of final reports for all patients, regardless ofage, undergoing bone scintigraphy that include physician documentationof correlation with existing relevant imaging studies (e.g., x-ray, MRI,CT, etc.) that were performed.

Various embodiments may provide the necessary support for clinicians athospitals and in other health care settings to better coordinate carefor patients beginning with radiographic digital imaging in a mannerthat will benefit patient care by objective measurements.

Of the approximate 129 PQRS measures currently in use, there are 24required radiological information standards to be shared betweenhospitals and physicians (see e.g., https://www.cms.gov/PQRS/); PublicLaw 109-432; Public Law 110-275). Radiological reports that must beshared include Computer Tomography (“CT”), Positron Emission Tomography(“PET”), Magnetic Resonance Imaging (“MRI”), Mammography, Plain Films(e.g., chest x-ray), and Bone Scintigraphy (i.e. bone scan) for avariety of specific purposes that include various cancers (9 measures),COPD (2 measures), Community Acquired Pneumonia (4 measures), Arthritis(2 measures), Osteopenia (4 measures), and Stroke (2 measures). The onlygeneral radiology PQRS measure is for a registry of all radiologyreports, both past and future, to be correlated to Bone Scintigraphy(bone scan) PQRS measure #147. It is widely accepted that many, if notall radiology imaging services, have a need to request and share bonescans from a variety of other imaging centers. Bone scans are routineprocedures from evaluation for fractures to a series of scanscoordinated with other imaging such as MRI, PET, CT, and plain films torule out metastases.

Hospital and physician administrators generally lack the technology andmethods to comply with PQRS Measure #147 in a scalable manner. Variousembodiments, as disclosed herein, address physician compliance inaddition to image sharing that provides shared savings to the physicianand credit for the participating site for meeting one PQRS measure. Bysupporting the national quality goal as defined by Centers for Medicareand Medicaid (“CMS”), embodiments of the present disclosure may assistthose health system administrators struggling to make their healthsystem IT infrastructure compliant to emerging and existent regulatoryrequirements. Various embodiments may also allow for expansion intoother quality projects involving radiology, especially in orthopedics,rheumatology, and cancer related conditions.

It is noted that table 900, and more specifically, QM codes, may beassociated with annotations (e.g., CPT codes) used with images. Forexample, with reference to FIGS. 6 and 12, the same principles describedabove for aggregator 404 may be applied. In this case, a melanomaexample may be used to produce an output that is a machine readableformat, such as table 900 of FIG. 12. Data inputs (e.g., images and/ortext), one or more processes, aggregation, semantics, ontologies andannotation may produce an output.

If the same principles are applied, collection and review of data inputs(e.g., images and/or text) may be desirable. In one specific example,one or more images may be of a skin screening (e.g., to look at molesand determine is a mole looks suspicious). If the mole is suspicious,the patient may be refereed to surgery (e.g., for excision of the lesionand so forth).

In one example, one or more diagnostic codes (e.g., classification ofdisease ICD-10 code) and/or one or more CPT codes may be linked to aquality measure code (e.g., PQRS 26, 27 138 and 194 and QOPI 1; see FIG.12). All of the above codes that comprise of billing for reimbursementand establishing quality are documented by all that takes place inaggregator 404 that is managed with a clinical diagnosis, which mayinclude clinical information.

According to one example, an output 416 of system 400 (see FIG. 6) maycomprise table 900. In this example, system 400 may process annotatedinputs (e.g., images and text) into a format defined by template module412 for consumption into an external system (EMR, LIS) or to an externalentity (e.g., Centers for Medicaid). The format may include theannotated FOV images that may include codes maintained by annotationservice module 410.

FIG. 13 illustrates a system 910 including a platform 912, in accordancewith an embodiment of the present disclosure. In addition to platform912, which may comprise a QM platform, system 910 includes a system(e.g., a laboratory information system or external system) 914, a HealthEconomics Optimization and Revenue Performance Management module (e.g.,an XIFIN® Health Economics Optimization and Revenue PerformanceManagement module) 916, an image-based QM library 918, a care teamcollaboration platform 920, an external system (e.g., an electronichealth records system, an electronic medical records system, or alaboratory information system) 912, and CMS 924. System 900 may alsoinclude another platform 926, which may also comprise a QM platform.

The output of the reporting system 900 may comprise the followingformats: 1) structured to be machine readable by larger enterprisesystems (EMR) and consumed by CMS and 2) rendered into a common formatfor human consumption. The system organization intelligence structureprocess may determine the necessary output for many destinations. Inputinto the system is collected from external billing, revenue cycle andpatient record systems, aggregated and organized into a structure typebased on parameters passed to system 900. It is noted that module 916,library 918, and/or platform 920 may comprise system 400, as illustratedin FIG. 6

FIG. 14 is a flowchart of a method 950, according to an embodiment ofthe present disclosure. Method 950 includes receiving a plurality ofdata inputs (act 952). Method 950 further includes establishing one ormore data sets including the plurality a plurality of data inputs,wherein one or more data inputs in each data set shares a commonidentifier (act 954). In addition, method 950 includes maintaining arelationship amongst data inputs in each data set (act 956). Moreover,method 950 includes generating one or more outputs based on data in adata set (act 958).

Embodiments of the disclosure may also include one or more systems forimplementing one or more embodiments disclosed herein. FIG. 15illustrates a schematic view of a processing system 970, according to anembodiment of the present disclosure. In an example, processing system970 may be integrated within any suitable computing device. Processingsystem 970 may include one or more processors 972 of varying coreconfigurations (including multiple cores) and clock frequencies.Processors 972 may be operable to execute instructions, apply logic,etc. It will be appreciated that these functions may be provided bymultiple processors or multiple cores on a single chip operating inparallel and/or communicably linked together. In at least oneembodiment, processors 972 may comprise and/or include one or more GPUs.

Processing system 900 may also include a memory system, which may be orinclude one or more memory devices and/or computer-readable media 974 ofvarying physical dimensions, accessibility, storage capacities, etc.such as flash drives, hard drives, disks, random access memory, etc.,for storing data, such as images, files, and program instructions forexecution by processors 974. In an embodiment, computer-readable media974 may store instructions that, when executed by processors 972, areconfigured to cause processing system 970 to perform operations. Forexample, execution of such instructions may cause processing system 970to implement one or more embodiments described herein.

Processing system 970 may also include one or more network interfaces976, which may include any hardware, applications, and/or othersoftware. Accordingly, network interfaces 976 may include Ethernetadapters, wireless transceivers, PCI interfaces, and/or serial networkcomponents, for communicating over wired or wireless media usingprotocols, such as Ethernet, wireless Ethernet, etc.

Processing system 970 may further include one or more peripheralinterfaces 978, for communication with a display screen, projector,keyboards, mice, touchpads, sensors, other types of input and/or outputperipherals, and/or the like. In some implementations, the components ofprocessing system 970 need not be enclosed within a single enclosure oreven located in close proximity to one another, but in otherimplementations, the components and/or others may be provided in asingle enclosure.

Memory device 974 may be physically or logically arranged or configuredto store data on one or more storage devices 980. Storage device 980 mayinclude one or more file systems or databases in any suitable format.Storage device 980 may also include one or more application programs982, which may contain interpretable or executable instructions forperforming one or more of the disclosed processes. When requested byprocessors 972, one or more of the application programs 982, or aportion thereof, may be loaded from storage devices 980 to memorydevices 974 for execution by processors 972.

Those skilled in the art will appreciate that the above-describedcomponentry is merely one example of a hardware configuration, as theprocessing system 970 may include any type of hardware components,including any necessary accompanying firmware or software, forperforming the disclosed implementations. Processing system 970 may alsobe implemented in part or in whole by electronic circuit components orprocessors, such as application-specific integrated circuits (ASICs) orfield-programmable gate arrays (FPGAs).

Although the foregoing description contains many specifics, these shouldnot be construed as limiting the scope of the disclosure or of any ofthe appended claims, but merely as providing information pertinent tosome specific embodiments that may fall within the scopes of thedisclosure and the appended claims. Features from different embodimentsmay be employed in combination. In addition, other embodiments of thedisclosure may also be devised which lie within the scopes of thedisclosure and the appended claims. The scope of the disclosure is,therefore, indicated and limited only by the appended claims and theirlegal equivalents. All additions, deletions and modifications to thedisclosure, as disclosed herein, that fall within the meaning and scopesof the claims are to be embraced by the claims.

What is claimed:
 1. A method comprising: receiving a plurality of imagesincluding a first image from a first medical image modality captured ata first point in time and a second image from a second medical imagemodality captured at a second point in time later than the first pointin time, the first medical image modality being different from thesecond medical image modality; generating a first visual annotation onthe first image and a second visual annotation on the second image, thefirst visual annotation on the first image including a first region ofinterest (ROI) on the first image, the generating of the first ROI onthe first image including identifying the first ROI on the first image,the second visual annotation on the second image including a second ROIon the second image, the first visual annotation and the second visualannotation applicable to a same medical condition, related medicalconditions, a lack of the same medical condition, or a lack of one orboth of the related medical conditions, and including a relationshiprepresentative of the applicability of the first visual annotation andthe second visual annotation to the same medical condition, relatedmedical conditions, a lack of the same medical condition, or a lack ofone or both of the related medical conditions; receiving the firstimage, the first visual annotation, the second image, and the secondvisual annotation by an aggregator; storing the first visual annotationas first metadata linked to the first image as a first vector ofinformation, and the second visualization as second metadata linked tothe second image as a second vector of information, the first image andthe second image stored in a same data set and stored in a manner tomaintain the relationship between the first visual annotation and thesecond visual annotation; based on a request involving the same dataset, recalling, by the aggregator, the first image, the first metadata,the second image, and the second metadata, for display, the firstmetadata recalled based on the first metadata being linked to the firstimage and the second metadata recalled based on the second metadatabeing linked to the second image; rendering and displaying, in a singleview, the first image with the first visual annotation simultaneouslywith the second image with the second visual annotation whilemaintaining the relationship, the first image disposed in a first regionof the single view and the second image disposed within the single viewin a second region physically separate from the first region of thesingle view, the single view further including a user interfaceincluding an annotation tool providing functionality by which a userelectronically adds one or more of text or markings to a given image;receiving user input invoking the annotation tool of the user interfaceto electronically apply a third visual annotation to at least one of thefirst image and the second image; and storing the third visualannotation based on the received user input as third metadata, includingtiming information relative to at least one of the first visualannotation and the second visual annotation.
 2. The method of claim 1,wherein: the first medical image modality is an ultrasound imagemodality, a progesterone receptor (PR) image modality, a magneticresonance imaging (MRI) image modality, a human epidermal growth factorreceptor (HER2) image modality, a positron emission tomography andcomputation tomography (PET/CT) image modality, a Hematoxylin-Eosin(H&E) image modality, an Immunohistochemistry (IHC) image modality, or aFluorescence In Situ Hybridization (FISH) image modality; and the secondmedical image modality is an ultrasound image modality, a PR imagemodality, an Mill image modality, an HER2 image modality, a PET/CT imagemodality; an H&E image modality, an IHC image modality, or a FISH imagemodality.
 3. The method of claim 1, wherein the first image and thesecond image are assigned and share a common patient identifier.
 4. Themethod of claim 1, wherein: the first image includes a first ontologicalcode; the second image includes a second ontological code; and at leastone of the first ontological code and the second ontological codeincludes a billing code.
 5. The method of claim 1, wherein thegenerating of the second ROI on the second image includes automaticallymapping the first ROI on the first image to the second image.
 6. Themethod of claim 1, further comprising: maintaining a second relationshipamongst the first image and the second image in the same data set via asemantic layer of a semantic ontological framework.
 7. The method ofclaim 6, further comprising: generating a report including one or moreoutputs based on the second relationship.
 8. The method of claim 7,wherein the one or more outputs include a continuity of care (CCD)structure output compliant with a standardized format.
 9. The method ofclaim 7, wherein the one or more outputs include a consolidated-clinicaldocument architecture (C-CDA) standard output.
 10. The method of claim7, wherein the one or more outputs include human-readable data.
 11. Themethod of claim 7, wherein the one or more outputs includemachine-readable data.
 12. The method of claim 1, wherein therelationship includes a field of view (FOV) relationship.
 13. One ormore non-transitory computer-readable media comprising one or morecomputer-readable instructions that, when executed by one or moreprocessors, cause the one or more processors to perform a method, themethod comprising: receiving a plurality of images including a firstimage from a first medical image modality captured at a first point intime and a second image from a second medical image modality captured ata second point in time later than the first point in time, the firstmedical image modality being different from the second medical imagemodality; generating a first visual annotation on the first image and asecond visual annotation on the second image, the first visualannotation on the first image including a first region of interest (ROI)on the first image, the generating of the first ROI on the first imageincluding identifying the first ROI on the first image, the secondvisual annotation on the second image including a second ROI on thesecond image, the first visual annotation and the second visualannotation applicable to a same medical condition, related medicalconditions, a lack of the same medical condition, or a lack of one orboth of the related medical conditions, and including a relationshiprepresentative of the applicability of the first visual annotation andthe second visual annotation to the same medical condition, relatedmedical conditions, a lack of the same medical condition, or a lack ofone or both of the related medical conditions; receiving the firstimage, the first visual annotation, the second image, and the secondvisual annotation by an aggregator; storing the first visual annotationas first metadata linked to the first image as a first vector ofinformation, and the second visualization as second metadata linked tothe second image as a second vector of information, the first image andthe second image stored in a same data set and stored in a manner tomaintain the relationship between the first visual annotation and thesecond visual annotation; based on a request involving the same dataset, recalling, by the aggregator, the first image, the first metadata,the second image, and the second metadata, for display, the firstmetadata recalled based on the first metadata being linked to the firstimage and the second metadata recalled based on the second metadatabeing linked to the second image; rendering and displaying, in a singleview, the first image with the first visual annotation simultaneouslywith the second image with the second visual annotation whilemaintaining the relationship, the first image disposed in a first regionof the single view and the second image disposed within the single viewin a second region physically separate from the first region of thesingle view, the single view further including a user interfaceincluding an annotation tool providing functionality by which a userelectronically adds one or more of text or markings to a given image;receiving user input invoking the annotation tool of the user interfaceto electronically apply a third visual annotation to at least one of thefirst image and the second image; and storing the third visualannotation based on the received user input as third metadata, includingtiming information relative to at least one of the first visualannotation and the second visual annotation.
 14. The one or morenon-transitory computer-readable media of claim 13, wherein: the firstmedical image modality is an ultrasound image modality, a progesteronereceptor (PR) image modality, a magnetic resonance imaging (MRI) imagemodality, a human epidermal growth factor receptor (HER2) imagemodality, a positron emission tomography and computation tomography(PET/CT) image modality, a Hematoxylin-Eosin (H&E) image modality, anImmunohistochemistry (IHC) image modality, or a Fluorescence In SituHybridization (FISH) image modality; and the second medical imagemodality is an ultrasound image modality, a PR image modality, an Millimage modality, an HER2 image modality, a PET/CT image modality; an H&Eimage modality, an IHC image modality, or a FISH image modality.
 15. Theone or more non-transitory computer-readable media of claim 13, whereinthe first image and the second image are assigned and share a commonpatient identifier.
 16. The one or more non-transitory computer-readablemedia of claim 13, wherein: the first image includes a first ontologicalcode; the second image includes a second ontological code; and at leastone of the first ontological code and the second ontological codeincludes a billing code.
 17. The one or more non-transitorycomputer-readable media of claim 13, wherein the generating of thesecond ROI on the second image includes automatically mapping the firstROI on the first image to the second image.
 18. The one or morenon-transitory computer-readable media of claim 13, further comprising:maintaining a second relationship amongst the first image and the secondimage in the same data set via a semantic layer of a semanticontological framework.
 19. The method of claim 18, further comprising:generating a report including one or more outputs based on the secondrelationship.
 20. The method of claim 19, wherein: the one or moreoutputs include a continuity of care (CCD) structure output compliantwith a standardized format, or the one or more outputs include aconsolidated-clinical document architecture (C-CDA) standard output. 21.The method of claim 1, wherein the first imaging modality generates thefirst image at a cellular level and the second imaging modalitygenerates the second image at a tissue level.
 22. The method of claim 1,wherein the first imaging modality generates the first image of tissuefrom a first direction and the second imaging modality generates thesecond image of the tissue from a second direction different than thefirst direction.